Experiments & Causal Inference: Why a Coin Flip Beats a Clever Model

For decades, doctors read the same pattern in large observational studies: women who took hormone therapy after menopause had fewer heart attacks. It was a big, careful body of research, and the advice followed it. Then the Women's Health Initiative did something simpler. It assigned thousands of women, by lot, to hormones or a placebo. The trial's combined-hormone arm was stopped early in 2002 because the treated women had more breast cancer and more cardiovascular events, not fewer (Rossouw and colleagues, JAMA, 2002). Researchers still discuss why the earlier studies pointed the other way. One often-cited explanation is that women who chose hormones were healthier and better off to begin with, in ways that adjusting for recorded factors did not fully remove.

That is the central worry of this subject. A pattern in data is a fact about the data. A cause is a claim about what would happen if you changed something, and no table of numbers contains that on its own. A graduate of a causal-inference course keeps a habit: before looking at an estimate, ask what would have happened to these same people without the treatment, and what in the design makes the comparison fair.

The tools are old and new. Ronald Fisher built the logic of randomised experiments at the Rothamsted agricultural station in the 1920s and wrote it up in The Design of Experiments (1935). The formal language of "what would have happened" came from Jerzy Neyman (1923) and Donald Rubin (1974), and Judea Pearl's diagrams (from the 1980s, in Causality, 2000) turned it into something you can draw. Today the same ideas run every A/B test at every large web company.

Aggregates can reverse: Simpson's paradox

In the fall of 1973, the University of California, Berkeley received 8,442 graduate applications from men and 4,321 from women. About 44% of the men and 35% of the women were admitted. It looks like proof of bias against women, and a lawsuit was threatened. Peter Bickel, Eugene Hammel and J. W. O'Connell looked inside (Science, 1975). Admissions are decided department by department, so they went there.

Take the six largest departments (anonymised as A to F in the paper). Together, 45% of the men and 30% of the women were admitted. But in four of the six departments, a higher share of women than men were admitted. The reason is where people applied. Women applied in large numbers to departments that rejected most applicants (in C to F, admission rates of 35% or less in the table below); men applied heavily to A and B, which admitted nearly two thirds. A fact about the whole reverses inside every part: Simpson's paradox.

Without JavaScript: pooled, 44.5% of 2,691 men and 30.4% of 1,835 women were admitted. In departments A, B, D and F women's rates were higher; in C and E men's were. If women had applied in the men's proportions, the same department rates would give 51.6%.

The tempting lesson, "always look at the subgroups", is wrong. The right lesson is that the data cannot tell you which view answers the question. If the question is "does a department discriminate in its decisions?", the department-level numbers are the right ones. If the question is "why do far fewer women end up in the programmes that admit most?", the pooled view and the earlier steps (who is encouraged to apply where) become the story. Which table to trust depends on the causal arrows, and the next sections give you a way to draw them. Bickel and colleagues concluded that the pooled gap came from where people applied, and that the department-level data showed, if anything, a small bias favouring women.

The missing counterfactual, and why a coin fixes it

The language that makes this precise is simple. Every person has two potential outcomes: how they would do if treated, and how they would do if not. Call them Y(1) and Y(0). The effect on that person is the difference. And we can never see both, because each person is treated or not: this is "the fundamental problem of causal inference" (Paul Holland's phrase, 1986). Every causal estimate fills in the missing half with a stand-in group, and the whole subject is about when the stand-in is fair.

The two agree when who gets treated has nothing to do with how people would have done anyway. They disagree when it does: if sicker people seek out the treatment, the treated group does worse on average, and the treatment looks harmful even when it helps each person. Below are 200 invented patients. The treatment truly raises everyone's recovery score by 6. Turn up how strongly the sicker patients choose treatment, then press "Peek" to see the column nobody gets to see in real life.

Without JavaScript: when sicker patients are much more likely to choose treatment, the treated group's average recovery is about 4 points below the untreated group's, although the treatment adds 6.

Randomisation breaks the link. If a coin decides who is treated, the two groups are alike in everything: severity, motivation, genes, luck, and things nobody measured. On average the treated-minus-untreated difference then equals the true average effect. "On average" matters: one experiment can still be off by chance, so repeat it and watch.

Without JavaScript: in 300 simulated studies at full selection, patient-chosen treatment gave an average estimate of -4.0 (5th to 95th percentile -5.5 to -2.3); coin-flip assignment gave 6.1 (3.7 to 8.2). The truth is 6.

Notice two things. The coin's estimates scatter around the truth, with the scatter set by sample size. The self-selected estimates scatter around the wrong number, and a bigger sample would only make them confidently wrong.

Draw the arrows first: confounders, mediators and colliders

Randomisation is the gold standard but often impossible or unethical: you cannot assign people to smoke. Then you rely on assumptions about how the data came to be, and Pearl's causal diagrams (directed acyclic graphs) make those assumptions visible. Each arrow says "this directly affects that". Three shapes cover most of what goes wrong:

• Confounder (a fork): something that affects both the treatment and the outcome. Hot weather raises ice-cream sales and drownings. Compare like with like, that is, adjust for it.
• Mediator (a chain): part of the route by which the treatment works. Exercise lowers resting heart rate partly by raising fitness. Adjust for it and you remove part of the effect you wanted; whether that is right depends on the question (total effect or direct effect).
• Collider (an inverted fork): something affected by both. Among famous actors, talent and looks can appear to trade off, simply because you need enough of one or the other to be famous. Adjusting for a collider, or selecting on it, creates a link that was not there. Joseph Berkson described the hospital version in 1946.

The surprising part is the last: adding a control variable can make an estimate worse. The rule is not "adjust for everything you have". It is "adjust for what the diagram says you should". Try each story below on simulated data in which the true answer is known.

Without JavaScript: in the heat story, the raw slope is about 0.63 and falls to about 0 after adjusting. In the fame story it is about 0 raw and about -0.8 after adjusting. In the exercise story the raw slope of -0.60 is the total effect, and adjusting leaves only the direct part, -0.2.

Experiments in practice: noise, power and peeking

Most experiments today are A/B tests. Half the visitors see version A, half see B, a coin decides, and you compare a metric such as the share who buy. The logic is exactly that of the previous section, so the worry is noise rather than bias. Suppose 10% of visitors buy, and the change really adds one percentage point. With 5,000 visitors per arm, the difference you observe wobbles by about 0.6 points from chance alone, so a one-point effect is only 1.7 wobbles tall. A standard rule of thumb (Lehr's formula, in van Belle's Statistical Rules of Thumb) says you need roughly this many per arm for 80% power at the 5% level:

Power is the chance that a real effect is detected. Underpowered tests miss real effects, and, because only the lucky big estimates reach significance, the winners they do find exaggerate the effect. The second trap is peeking. If you look at the running result every day and stop the first time p dips under 0.05, you are giving chance many tries. With ten looks, a test with no true effect declares a winner about 19% of the time (in the run below), not 5%. Sequential tests exist that allow peeking honestly, and are what large experimentation platforms use. A third trap is the number of metrics: with 20 independent metrics and no real effect, the chance that at least one shows p below 0.05 is 1 − 0.95²⁰, about 64%.

Without JavaScript: with a true lift of 0, 400 experiments of 1,000 visitors per arm give a "winner" about 4% of the time if you look once, and about 19% if you peek ten times. With 5,000 per arm and a true one-point lift, about 37% win with a single look.

Interference: when your treatment changes the control

Everything above quietly assumed that one person's treatment does not affect another's outcome. Rubin called this the "stable unit treatment value assumption". It fails whenever units compete or talk. Vaccinate half a town and the unvaccinated are protected too. Give half the sellers on a marketplace a discount and they take sales from the other half. Show half of a social network's users a new feed and their friends in the control group are affected.

The marketplace case is the clearest. Suppose 100 sellers share a fixed pool of 1,000 buyers. In a test, a randomly chosen share gets a discount that makes them more attractive. The test compares treated sellers with control sellers and reports a big lift. But much of it was taken from the controls. If everyone got the discount, nothing would change in a fixed market. Engineers meet this in two-sided platforms, ride-hailing and ad auctions, and respond with experiments randomised by whole cities, regions or time blocks, which cost power but keep the arms from touching.

Without JavaScript: with half the sellers discounted at 1.5 times as attractive and no extra demand, treated sellers sell 12 a day and control sellers 8. The test reports +50%. Giving everyone the discount changes nothing, so the true roll-out effect is 0%.

When you cannot randomise: adjust for what you measured, fear what you did not

With observational data, the standard move is to compare like with like on everything you recorded: split people into groups with similar severity (strata), match them, or model the chance of treatment from their characteristics (a propensity score, Rosenbaum and Rubin, 1983). If every factor that drives both treatment and outcome was measured, this works. The fatal catch is that you can never check that "if". The question to ask of every observational claim is: what unmeasured thing might have pushed people into the treatment and changed their outcome? Analysts now quantify how strong such a hidden factor would have to be to erase a result, with tools such as the E-value (VanderWeele and Ding, 2017).

Where a rule or an accident acts like a coin flip, you can do better than adjustment: a policy that hit one state but not its neighbour (difference-in-differences), a lottery that nudged people (instrumental variables), a cutoff that put near-identical people on either side (regression discontinuity), or a weighted blend of untreated places built to match the treated one (synthetic control, Abadie and colleagues). The 2021 economics Nobel prize went to David Card, and to Joshua Angrist and Guido Imbens, for this credibility revolution; Econometrics (34/44) works through three of them. Try the first idea here, and then the limit.

Without JavaScript: with no hidden factor, the raw difference is about -1.1 (the treatment looks harmful), and adjusting for severity gives about 4.7 against a truth of 5. With a hidden factor of strength 5, adjusting gives about -2.8.

What a graduate just sees

• A causal claim compares what happened with what would have happened, and the second is never observed.
• A pooled number and a within-group number can point opposite ways; the causal story decides which answers the question.
• Randomisation is powerful because it balances what nobody measured, not because it is a ritual.
• Draw the arrows before the regression: adjust for confounders, think twice about mediators, never for colliders.
• Adding more controls is not always safer, and an unmeasured confounder is a risk that no sample size removes.
• An A/B test is a noisy instrument: ask for the sample size, the number of looks and the number of metrics.
• If units can affect each other, the treated-versus-control gap is not the effect of a roll-out.
• Effects differ across people; an average can hide who gains and who loses, and the right answer depends on who is being asked about.

Where it shows up next

• Econometrics (34/44): the natural-experiment toolkit, applied to prices, schooling and minimum wages.
• Research Methods in Psychology (35/19): experiments, effect sizes, power and the replication crisis are this piece's tools in another field.
• Data Ethics, Privacy & Fairness (36/37): "did the algorithm cause this disparity?" is a causal question, and fairness definitions differ in their causal assumptions.
• In the world: drug trials and vaccine studies, product teams at every large web company, development economics (randomised trials of schooling and cash transfers, Nobel 2019), and policy evaluation by governments.

If you want to go deeper

• 📘 Textbook. Miguel Hernán and James Robins, Causal Inference: What If (Chapman & Hall/CRC, 2020), free from Hernán's Harvard page, rigorous and diagram-based; and Guido Imbens and Donald Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences (Cambridge University Press, 2015), for potential outcomes.
• 📘 For experiments. Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020), written from running experiments at large technology companies.
• 📘 Gentler. Judea Pearl and Dana Mackenzie, The Book of Why (2018), and Joshua Angrist and Jörn-Steffen Pischke, Mastering 'Metrics (Princeton, 2015), a short, friendly tour of the econometric toolkit. Scott Cunningham's Causal Inference: The Mixtape (Yale University Press, 2021) is also available to read free online.
• 🌐 Free course. MIT OpenCourseWare 14.387 Applied Econometrics: Mostly Harmless Big Data (Fall 2014, Angrist and Chernozhukov), a graduate-level course whose notes and problem sets are public.